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Field of the invention 
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The present invention relates to a vehicle surroundings 



7 monitoring apparatus and more in particular to a vehicle 

8 surroundings monitoring apparatus suitable for detecting solid 

9 objects successively disposed along roads such as guardrails, 
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side walls and the like. 
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2 . Prior arts 




12 


With increased number of vehicles and with increased 


Issf ' 
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number of traffic accidents, the conception of Advanced Safety 
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Vehicle (ASV) is becoming one of primary strategies in designing 
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a vehicle. In particular, an ASV technology raises the 


fx! 
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intelligence level of a vehicle through the state of the art 
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electronic technologies. In recent years, various safety devices 


i y 
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such as issuing an alarm to inform a vehicle driver of a likelihood 
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of collision, stopping a vehicle, decreasing or increasing the 



^ 20 speed of the vehicle by detecting a vehicle traveling ahead or 

21 an obstacle through a television camera, a laser-beam radar or 

22 the like, have been proposed. 



24 Patent Application Laid-open No. Toku-Kai-Hei 5-265547, have 

25 already disclosed a technique in which images taken by two 

26 stereoscopic cameras are transformed into distance images, these 

27 distance images being divided into lattice-like small regions 

28 at a prescribed interval to detect solid objects for each small 
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The applicant of the present invention, in Japanese 



l 



1 region. Furtner, the applicant, in Japanese 'Patent Application 

2 Laid-open No. Toku - Kai - Hei 6-266828, have already proposed a 

3 technique wherein similarly data of solid objects per small 

4 region are extracted, these data being processed by a so called 

5 "Hough" transformation method to detect the solid objects such 

6 as side walls and guardrails aligned along roads. 

7 However, according to these known arts, since the data 

8 of solid objects are processed by the "Hough" transformation 

9 method and the like, as a result, with respect to the solid objects 

10 provided along a curved road like guardrails, only their small 

11 portions located within a relatively short distance are 

12 recognized as straight lines, therefore it is much more difficult 
Q 13 to recognize those objects in the distance. 

fij 14 

§J 15 SUMMARY OF THE INVENTION 

III 16 The present invention is intended to obviate the 

Lk 17 aforesaid problem of the prior arts and it is an object of the 

ry 18 present invention to provide a vehicle surroundings detecting 

Jfi 19 apparatus capable of detecting a series of solid objects which 

^ 20 constitute a boundary of a road as a wall surface even in case 

21 where the road is curved. 

22 In order to achieve the object, the present invention 

23 comprises a wall surface detecting means for dividing positional 

24 data of solid objects into groups and based on the grouped 

25 positional data of the solid objects for detecting a wall surface 

26 formed along a boundary of a road, a wall surface model forming 

27 means for interconnecting a plurality of nodes and based on the 

28 interconnected nodes for forming a wall surface model to express 



1 an outline of the side wall arid a wall surface model correcting 

2 means based on the grouped positional data of the solid objects 

3 for correcting the wall surface model. 
4 

5 BRIEF DESCRIPTION OF THE DRAWINGS 

6 Fig. 1 is an overall view of a vehicle surroundings 

7 detecting apparatus mounted on a vehicle; 

8 Fig. 2 is a schematic block diagram of a vehicle 

9 surroundings detecting apparatus according to the present 

10 invention; 

11 Fig. 3 is a first flowchart showing a flow of control 

12 of a solid object/side wall group detecting process; 

O 13 Fig. 4 is a second flowchart showing a flow of control 

pj 14 of a solid object/side wall group detecting process; 

m 15 Fig. 5 is a third flowchart showing a flow of control 

jjj 16 of a solid object/side wall group detecting process; 

J\ 17 Fig. 6 is a flowchart showing a flow of control of a 

18 wall surface detecting process; 

y 19 Fig. 7 is a first flowchart showing a flow of control 

=3=7. 

^ 20 of a wall surface position correcting process; 

21 Fig. 8 is a second flowchart showing a flow of control 

22 of a wall surface position correcting process; 

23 Fig. 9 is an explanatory view showing an example of 

24 images taken by cameras mounted on a vehicle; 

25 Fig. 10 is an explanatory view showing an example of 

26 distance images shown in Fig. 9; 

27 Fig. 11 is an explanatory view showing the position 

28 of solid objects detected per respective strips; 
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1 Pig. 12 is an explanatory view showing the result of 

2 detection of side walls; 

3 Fig. 13 is an explanatory view showing the result of 

4 detection of side walls in terms of the X-Z plane; 

5 Fig. 14 is a schematic view showing a wall surface 

6 model; 

7 Fig. 15 is an explanatory view showing the way of 

8 searching a wall surface pattern; 

9 Fig. 16 is an explanatory view showing an example of 

10 a pattern of a weight coefficient; 

11 Fig. 17 is an explanatory view showing the result of 

12 calculation of a degree of coincidence; 

y 13 Fig. 18 is an explanatory view showing the connection 

[il 14 of nodes; 

SI 15 Fig. 19 is an explanatory view showing the result of 

fU 16 detection of wall surfaces; and 

17 Fig. 20 is an explanatory view showing the result of 

jffj 18 detection of wall surfaces in terms of the X-Z plane. 

5 19 

2 0 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

21 Referring now to Fig* 1, reference numeral 1 denotes 

22 a vehicle on which a vehicle surroundings monitoring apparatus 

23 2 is mounted for imaging objects within a visible scope ahead 

24 of the vehicle and for recognizing the objects for monitoring. 

25 The vehicle surroundings monitoring apparatus 2 comprises a 

26 stereoscopic optical system 10 for imaging objects from two 

27 different positions, an image processor 20 for processing images 

28 of these objects to obtain three-dimensional distance 
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CTiformation, and a recoanition/ii 



1 distribution Wf ormation, and a recognition/judgment' computer 

2 30 for detecting three-dimensional positions of roads and solid 

3 objects at high speeds based on the distance information inputted 

4 from the image processor 20, for identifying a preceding vehicle 

5 or an obstacle based on the result of the detection and for judging 

6 whether or not an alarm should be issued to avoid a collision 

7 with the preceding vehicle or the obstacle. 

8 The recognition/judgment computer 30 is connected with 

9 sensors such as a vehicle speed sensor 4, a steering angle sensor 

10 5 and the like in order to detect a present traveling condition 

11 of the vehicle and also it is connected with a display 9 provided 

12 at the front of a vehicle driver for informing hazard. Further, 
□ 13 the computer 30 is connected with an external interface for 
U 14 example for controlling actuators (not shown) which operate so 
n 15 as automatically to avoid a collision with the obstacle or the 

; 16 vehicle traveling ahead. 
iU: 17 The stereoscopic optical system 10 is composed of a 

18 pair of left and right CCD (Charge Coupled Device) cameras 10a, 

19 10b. A pair of stereoscopic images taken by the CCD cameras 10a, 

20 10b are processed in the image processor 20 according to the 

21 principle of triangulation to obtain three-dimensional distance 

22 distribution over an entire image. 

23 The recognition/ judgment computer 30 reads the 

24 distance distribution information from the image processor 20 

25 to detect three-dimensional positions with respect to the 

26 configuration of roads and solid objects such as vehicles and 

27 obstacles at high speeds and judges a possibility of collision 

28 or contact with these detected objects based on the traveling 
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condition det 



ed by the vehicle speed sensor 4 and the- steering 



2 angle sensor 5 of the self vehicle to inform the vehicle driver 

3 of the result of the judgment through the display 9. 



5 and the recognition/ judgment computer 30. The image processor 

6 20 comprises a distance detecting circuit 20a for producing 

7 distance distribution information and a distance image memory 

8 20b for memorizing this distance distribution information. More 

9 specifically, the distance detecting circuit 20a calculates a 

10 distance to a given object by selecting a small region imaging 

11 an identical portion of the object from the left and right 

12 stereoscopic images taken by the CCD cameras 10a, 10b, 
W 13 respectively and then obtaining a deviation between these two 
HJ 14 small regions and outputs in the form of three-dimensional 
01 15 distance distribution information. 



H 17 the left and right CCD cameras 10a, 10b. When this image is 

fU 18 processed by the distance detecting circuit 20a, the distance 

O 19 distribution information outputted from the distance detecting 

20 circuit 20a is expressed as a distance image as shown in Fig. 

21 10 . 

22 The example of the distance image shown in Fig. 10 has 

23 a picture size composed of 600 (laterally) x 200 (longitudinally) 

24 picture elements. The distance data are included in white dotted 

25 portions that correspond to the portions having a large difference 

26 of brightness between two adjacent picture elements aligned in 

27 the left and right direction respectively in the image shown in 

28 Fig. 9. Further, in this example, the distance detecting circuit 



Fig. 2 shows a constitution of the image processor 20 



ry . i6 



Fig. 9 shows an example of either of images taken by 
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1 20a treats distance image as an .image composed of 150 

2 (laterally) x 50 (longitudinally) blocks, i.e., 4x4 picture 

3 elements for one block or one small region. The calculation of 

4 distance is performed for each block of the left and right images. 

5 The recognition/ judgment computer 30 comprises a 

6 microprocessor 30a primarily for detecting the road configuration, 

7 a microprocessor 30b primarily for detecting solid objects based 

8 on the configuration of a road detected and a microprocessor 30c 

9 primarily for identifying a preceding vehicle or an obstacle based 

10 on the positional information of the detected solid objects and 

11 for judging a possibility of collision or contact with the 

12 preceding vehicle or the obstacle and these microprocessors 30a, 
g 13 3 0b, 30c are connected in parallel with each other through a system 
5 14 bus 31. 

J is The system bus 31 is connected with an interface circuit 

KJ 16 32 to which the distance image is inputted from the distance image 

L 17 memory 20b, a ROM 33 for storing a control program, a RAM 34 for 

IK 18 memorizing miscellaneous parameters produced during 

19 calculations, an output memory 35 for memorizing the result of 

20 processing, a display controller 30d for controlling the display 

21 9 and an I/O interface circuit 37 to which signals are inputted 

22 from the vehicle speed sensor 4 and the steering angle sensor 

23 5 . 

24 As shown in Fig. 9, the distance image has a coordinate 

25 system composed of a lateral axis i, a longitudinal axis j and 

26 a vertical axis dp with an origin of the coordinates placed at 

27 the left below corner of the distance image. The vertical axis 

28 dp indicates a distance to an object which corresponds to the 
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1 deviation amount between blocks. Therefore, in the image 

2 processing computer 30, a point (i, j, dp) on the distance image 

3 is transformed into a coordinate system provided in the real space 

4 to perform processes such as recognition of the road configuration, 

5 detection of the position of solid objects and the like. 

6 That is to say, with respect to the three-dimensional 

7 coordinate system fixed to a self vehicle in the real space, 

8 setting X axis on the right side with respect to the traveling 

9 direction of the self vehicle (vehicle 1) , Y axis in the upward 

10 direction of the vehicle 1 and Z axis in the forward direction 

11 of the vehicle and placing an origin of the coordinates on the 

12 road surface underneath the center of two CCD cameras 10a, 10b, 
M 13 X-Z plane (Y = 0) coincides with the road surface, if the road 

14 is flat. Accordingly, the point (i, j, dp) on the distance image 

15 can be transformed into a point (x, y, z) in the real space as 
ITU 16 follows: 

H 17 x - CD / 2 + z ' PW - (i - IV) (1) 

ry 

HI 18 y - CH + z • PW • (j - JV) (2) 

Q 

Q 19 z = KS / dp (3) 

20 where CD is an interval between CCD cameras 10a, 10b; PW is an 

21 angle of visibility per picture element; CH is a height of CCD 

22 cameras when measured from the road surface; IV, JV are 

23 coordinates of an infinite point directly in front of the vehicle 

24 1 on the image; and KS is a distance coefficient (KS ■ CD / PW) . 

25 Rewriting the above equations (1), (2) and (3) as 

26 follows: 

27 i = (x -CD / 2) / (z ' PW) + IV (4) 

28 j - (y - CH) / (z • PW) + JV (5) 
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1 dp - KS / z -....( 6 ) 

2 Next, processes in the recognition/ judgment computer 

3 30 will be described. 

4 In the microprocessor 30a, first, actual lane markers 

5 of a road are extracted from three-dimensional positional 

6 information contained in the distance image which is stored in 

7 the distance image memory 20b and then the configuration of the 

8 road is recognized by modifying parameters of a built-in road 

9 model so as to agree with the actual road configuration. 

10 The road model described above is expressed by a 

11 plurality of three-dimensional linear equations. That is, the 



12 imaged left and right lane markers of the road on which the subject 

5! 13 vehicle is traveling are divided into a plurality of intervals 

H 14 determined according to distance and the road model is formed 

| 15 by a plurality of broken lines each of which is expressed for 

W 16 every interval in the following three-dimensional linear 

H* 17 equations: 

Oj 18 x « a ' z + b ....(7) 

O 1 9 y=c*z+d ....(8) 

2 0 where a, b are parameters of a linear equation extended in the 

21 horizontal direction in the coordinate system of the real space 

22 and c, d are parameters of a linear equation extended in the 

23 vertical direction in the coordinate system of the real space. 



24 in the microprocessor 30b wherein the detection of 

25 solid objects is processed, the distance image is divided into 

26 lattice- like strips having a prescribed interval and data of 

27 solid objects are extracted for every strip. Then, a histogram 

28 is produced per each of these strips based on the data of solid 



9 



1 objects and the position on the X-Y plane of solid objects 

2 representing respective strips and the distance thereto are 

3 obtained from the histogram. Then, comparing the image 

4 successively from the left to the right, the images having close 

5 distances in the forward and backward direction (Z-axis 

6 direction) and the lateral direction (X-axis direction) are 

7 classified into the same group. Further, when the arrangement 

8 direction of the data is checked, the portion where the 

9 arrangement direction is largely changed is found, the group 

10 being divided in a different group. 

11 Further, based on the arrangement direction of the 

12 distance data of the overall groups, i.e., the gradient with 

13 respect to the Z-axis, the groups are classified into solid object 
pf 14 groups or side wall groups. For the solid object groups, 
m is parameters such as a mean distance, X coordinates of the left 

16 and right ends and the like are calculated from the distance data 

M 17 of the group. Further, for the side wall groups, parameters such 

fy 18 as the arrangement direction (gradient with respect to the Z-axis) , 

o * 

Q 19 the positions of the forward and backward ends in terms of Z-X 

20 coordinates and the like are calculated. Thus, the front end, 

21 the side surface and the rear end of a solid object and the 

22 structure such as a guardrail are detected as the side wall 

23 arranged along the road. 

24 With respect to the generation of distance image, the 

25 process of detecting the configuration of roads from the distance 

26 image and the process of the judgment of collision or contact 

27 with obstacles/ details of which are described in Japanese Patent 

28 Application Laid-open No. Toku-Kai-Hei 5-265547 and No. 
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1 Toku-Kai-Hei 6-266828 .both of which have been proposed by the 

2 applicant of the present invention. 

3 The present invention is characterized in that even 

4 when the road is curved, the wall surface can be recognized up 

5 to the far distance along the curved road. The process in the 

6 microprocessor 30b will be described according to the flowcharts 

7 shown in Fig. 3 through Fig. 8. 

8 The programs shown in Figs 3 through 5 are ones for 

9 classifying the solid object group and the side wall group by 

10 processing the distance data obtained from the distance image. 

11 First, at a step S101, the distance image is divided into 

12 lattice-like strips having a given interval (for example 8 to 

13 20 picture elements) and at S102 data of a solid object are 

14 extracted for every strip and the data of the first strip are 

15 read to calculate the distance to the object. 

16 Next, the program goes to S103 where the data of the 

17 first strip are set and at S104 the three-dimensional position 

18 (x, y, z) of the object is obtained according to the aforesaid 

19 equations (1) to (3) . Then, the program goes to S105 where the 

20 height yr of the road surface at the distance z is calculated 

21 according to the linear equations (7) and (8) expressing the road 

22 configuration. In case where the road configuration can not be 

23 recognized for example on a road having no lane marker, the road 

24 surface being assumed to be in a horizontal relation with the 

25 vehicle 1, the road height is established to be zero for example. 

26 Next, the program goes to S106 where the data above 

27 the road surface are extracted as the solid object data based 

28 on the height H from the road surface which is calculated according 
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1 to the following equation (9) . 

2 H - y - yr (9) 

3 In this case, if the height H of the object is 0 . 1 meters or smaller, 

4 since the object of this size is supposed to be a lane marker, 

5 a stain or a shadow on the road, the data of the object are 

6 discarded. Similarly, since the object which is located at a 

7 position higher than the self vehicle 1 is supposed to be a bridge 

8 or a signpost, that object is discarded. Thus, only the data of 

9 objects which are estimated to be solid objects on the road are 

10 selected . 

11 After that, the program goes to S107 where it is checked 
y . 12 whether or not the data is final one of the strip. If the data 
JJ' 13 is not final, after the next data is set at S108, the program 
J^rf 14 returns to S104 and similar processes are repeated to extract 

15 the data above the road surface. Further, when the final data 

W 16 of the strip are finished to be processed, the program goes from 

H 17 S107 to S109 wherein a histogram is prepared. The histogram is 

Pj 18 composed of a number of data contained within a predetermined 

O 19 interval of the distance z which is aligned on the lateral axis. 

20 At the next step S110, if there is an interval in which 

21 the frequency (number of data) is above a threshold value and 
2 2 further indicates a maximum value, it is judged that a solid obj ect 

23 exists within that interval and the distance to the object is 

24 detected. Thus prepared histogram also contains data erroneously 

25 detected and therefore some data appear in the position where 

26 no object exists. However, it should be noted that if there is 

27 an object having some degrees of size in a position, the frequency 

28 at the position shows a relatively large value and if there is 
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1 no object, the frequency is relatively • small , 

2 Accordingly, it is permissible to judge that if the 

3 frequency of the histogram exceeds a predetermined threshold 

4 value and besides shows a maximum value in an interval, an object 

5 exists in the interval and that if the maximum value of the 

6 frequency is below the threshold value, no object exists. Even 

7 in case where some amount of noises are included in the image 

8 data, it is possible to detect an object with minimum effect of 

9 noises. 

10 After that, the program goes from Sill to S112 where 

11 it is checked whether or not the process has reached a final strip. 

12 If it is judged that the process has not yet reached a final strip, 

13 the program returns to S103 and similar processes are repeated. 

14 When the process reaches the final strip, the program goes from 

15 S112 to S114. 

16 Fig. 11 is a view showing the position of solid objects 

17 detected for each strip from the original image. The distance 

18 data of these solid objects are classified into groups having 

19 a close distance with each other by the processes executed at 

20 the steps S114 to S120 . The grouping will be performed as follows . 

21 In these processes, the detected distances of the solid objects 

22 in respective strips are investigated. If the difference of the 

23 detected distances to the solid objects between adjacent strips 

24 is smaller than a threshold value, these objects are deemed to 

25 be the same objects and on the other hand, if that difference 

26 exceeds the threshold value, those objects are regarded as 

27 different objects. 

28 Specifically, at S114, the first strip (for example, 
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1 a strip of the left end) is investigated* and if a solid object 

2 is detected therein, the distance data are read and this strip 

3 Rl is classified into a group Gl having a distance Zl. Next, the 

4 program goes to S115 where the right adjacent strip R2 is 

5 investigated. If no solid object is detected in the strip R2 , 

6 it is judged that the group Gl exists within the strip Rl or in 

7 the neighborhood thereof and the distance is Zl . On the other 

8 hand, if a solid object is detected in the strip R2 and the distance 

9 to the object is Z2, the distance Zl of the strip Rl is compared 

10 with the distance Z2 of the strip R2 . 

11 After that, the program goes to S116 where it is judged 

12 whether or not the difference between the distances Zl and Z2 

13 is smaller than a threshold value and if the difference is smaller 

14 than the threshold value and is close to each other, it is judged 

15 at S117 that the solid object detected in the strip R2 belongs 
M 16 to the same group Gl to label as such and then the program goes 

17 to S119 . At this moment, the distance to the object is established 

18 to be a mean value of Zl and Z2 . 

19 On the other hand, in case where the difference of the 

20 distances Zl and Z2 exceeds the threshold value, the program goes 

21 from S116 to S118 in which, judging that the solid object detected 

22 in the strip R2 does not belongs to the group Gl, the solid object 

23 is labeled as belonging to a new group G2 having a distance Z2 

24 and then goes to S119. 

25 At S119, it is investigated whether or not the process 

26 has reached a final strip and if not, after the distance of the 

27 next strip is read at S120, the program returns to S115 and further 

28 the right adjacent strip is investigated. If the process has 
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1 reached the final strip, the program goes from S119 to S121. 

2 The following case should be noted. Assuming the 

3 situation where a vehicle parks beside a guardrail, there is a 

4 possibility that the distance data of the guardrail are deemed 

5 to belong to the same group as the distance data of the parked 

6 vehicle. In order to avoid this, the arrangement direction of 

7 the distance data is checked on the X-Z plane through the processes 

8 at S121 to S131 to divide the group of the arrangement direction 

9 into a portion in parallel with Z-axis and a portion in parallel 

10 with X-axis. 

11 That is, at S121 the data of the first group are read 

12 and at S122 the arrangement direction of the data of the respective 

13 strips is calculated. Further, at S123 these strips are labeled 

14 as "object" or "side wall", respectively. Specifically, first 

15 two points on the X-Z plane are picked up from the data of the 

16 first group. One point (XI, Zl) is a middle point of a strip Kl 

17 at the left end of the first group and the other point (Xp, Zp) 

18 is a middle point of a strip far away from the left end strip 

19 Kl by an interval of N strips in the right hand direction. Then, 

20 a line connecting these two points is drawn on the X-Z plane and 

21 a gradient Al of the line is calculated. When the gradient Al 

22 is compared with a prescribed value, for example 45 degrees, if 

23 the gradient Al is larger than the value, the strip Kl is labeled 

24 as "side wall" and if the gradient Al is smaller than the value, 

25 the strip Kl is labeled as "object". 

26 The interval N between strips is preferably N - 2 to 

27 4 . The reason is that N = 1, namely, an adjacent strip may produce 

28 fluctuations in the arrangement direction of the data due to the 
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1 dispersion of detected distance and as a "result it becomes 

2 difficult to make discrimination between "side wall" and "obj ect" . 

3 Therefore, it is suitable to use not an adjacent strip, but a 

4 strip a little distant. Hereinafter, the labeling of "side wall" 

5 or "object" is performed successively from the left end strip 

6 up to the strip apart by N strips on the left side of the right 

7 end strip. 

8 When the labeling is accomplished for each strip of 

9 the group, the program goes from S123 to S124 where the label 

10 of the left end strip is read and at the next step S125, the label 

11 of the right adjacent strip is read. Then, it is investigated 

12 whether or not the label of the left end strip is different from 
Q 13 that of the right adjacent strip. As a result, if the label of 
fjj 14 the left end strip is the same as that of the right adjacent strip, 
S 15 the program skips to S128 and if different, the program steps 
m 16 to S127 where the strip labeled "side wall" and the strip labeled 

17 "object" are divided into different groups respectively. The 
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18 division of the group is performed at the position apart by N/2 

19 strip on the right side of the position where the label changes 

20 from "side wall" to "object" and vise versa. 

21 * In this case, to avoid the situation where the label 

22 itself is erroneously labeled due to the dispersion of distance 

23 data, the division is performed only when more than three same 

24 labels are successive. 

25 At S128, it is checked whether or not the process comes 

26 to the final strip and if not, after reading the label of the 

27 next strip at S129, the program returns to S125 and hereinafter 

28 similar processes are repeated. When the process comes to the 
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final strxj^^'he program goes from S128^xo S130 where it is 
investigated whether or not the process reaches the final group. 
When the process does not yet reach the final group, the data 
of the next group are read and hereinafter the same processes 
are carried out repeatedly. When the process reaches the final 
group, the division of the groups is completed and the program 
goes from S130 to S132. 

The following steps S132 to S137 are of processes in 
which further classifications of "side wall" or "object" are 
carried out to raise the accuracy of the classification performed 
at S127. After the data of the first group are read at S132, at 
S133 approximate straight lines are obtained from the positions 
(Xi, Zi) within the group according to the Hough transformation 
or the linear square method to calculate a gradient overall the 
group . 

Then, the program goes to S134 where the group is 
reorganized such that the group having a gradient inclined toward 
X-axis is classified into the "object" group and the group having 
a gradient inclined toward Z-axis is classified into the "side 
wall" group. Further, at S135, miscellaneous parameters of the 
group are calculated. With respect to the group classified 
"object", these parameters include an average distance which is 
calculated from the distance data within the group, X-coordinates 
at the left and right ends of the group and the like and with 
respect to the group classified "side wall", those parameters 
include an arrangement direction of the data (gradient with 
respect to Z-axis) , Z, X coordinates of the front and rear ends 
of the group and the like. In this embodiment, in order to raise 
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1 the accuracy of classification, the fcrroup is reclassified 

2 according to the calculated gradient of the overall group, however 

3 this reclassification may be omitted. 

4 Further, the program goes from S135 to S136 where it 

5 is judged whether or not the process has reached the final group. 

6 If it is not the final group, the program goes back to S137 in 

7 which the data of the next group are read and returns to S133 

8 to repeat the same processes. When the process has reached the 

9 final group, the program leaves the routine. 
Fig. 12 shows a result of the detection of the side 

11 wall . When the data of the groups are illustrated on the X-Z plane, 

12 as shown in Fig. 13, they are recognized as "side wall" groups. 

13 In this case, portions along a curved road are not recognized. 

14 The program shown in Fig. 6 is for recognizing the wall surface 

15 along the curved road using the data of the "side wall" group 

16 obtained by the program described before. 

17 First, at S201, groups estimated to be a wall surface 
are selected from the groups classified "side wall" and at the 

O 19 steps after S202, a wall surface is searched based on the data 

20 of the "side wall" groups using the following wall surface model. 

21 The wall surface model is shown in Fig. 14, in which 

22 the wall surfaces are expressed as border lines connecting between 
nodes provided at a specified interval within a given range. For 
example, the border line is constituted by 41 nodes arranged at 

25 an interval of 2 meters within an range from 10 to 90 meters ahead 

26 of the self vehicle. Respective nodes have successive reference 

27 numbers starting from the self vehicle side. The Z - coordinates 

28 of the respective nodes are fixed with respect to the vehicle 



'~i 18 



23 
24 



18 



1 position ancMRie X - coordinates thereof are determined according 
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2 to the procedure which will be described hereinafter. 

3 At S202, a node N a corresponding to an end point on the 

4 vehicle side of the selected side wall group is established based 

5 on the z-coordinate of the end point and the X-coordinate of the 

6 node N s is established being adjusted to the X-coordinate of the 

7 end point. Next, the program goes to S203 where the next node 

8 N a+ i is established in the direction of the gradient of the side 

9 wall group. Next, when the node N a *i (i^2) is determined, its 

10 direction is established along a direction of the second previous 

11 node. 

12 Then, the program goes to S204 where, as shown in Fig. 

13 15, the position of the wall surface is searched by a so-called 

14 "pattern matching" within a specified searching range to extract 

15 a solid object Pi for every strip within the searching range. For 

16 example, the searching range in the X-axis direction has ± 3 to 

17 5 meters in the X-axis direction and ± 1 meter in the Y-axis 
I 18 direction with its center placed at a coordinate {X nQ + it Z ns u)of 

19 the node N s+ i established at S203 . 

20 The matching of the wall surface pattern is performed 

21 to the solid object Pi within the searching range. Fig. 16 shows 

22 an example of the wall surface pattern (weight coefficient 

23 pattern) used for the pattern matching. The wall surface pattern 

24 shown in Fig. 16 is a pattern for the wall surface on the left 

25 side and a symmetric pattern to this pattern is used for the wall 

26 surface on the right side. The lateral axis of this wall surface 

27 pattern coincides with the distance in the X-axis direction and 

28 the longitudinal axis indicates a weight coefficient. A maximum 
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1 point of a degree of coincidence is sear-ched while the central 

2 point of the wall surface pattern is shifted towards the X-axis. 

3 Specifically, as shown in Fig. 17, a weight Wi is obtained with 

4 respect to the deviation of the solid object Pi in the X-axis 

5 direction from the central point of the wall surface pattern to 

6 calculate the sum of the weight Wi as a degree of coincidence F. 

7 Further, when the degree of coincidence F becomes maximum, the 

8 position of the central point of the wall surface pattern is 

9 recognized as a wall surface . When the maximum value of the degree 

10 of coincidence F is smaller than a threshold value, it is judged 

11 that there is no wall surface. 

12 When the process of the step S204 finishes, the program 

13 goes to S205 at which a X-coordinate x pw of the central point of 

14 the wall surface pattern at the maximum point of the degree of 

15 coincidence F, is established as a X-coordinate of the wall 

16 surface pattern corresponding to the node N 8+ i. 

17 Then, the program goes to S206 where it is checked 

18 whether or not the node is the last one of the side wall group 

19 selected and if it is not the last node, the program goes back 

20 to S203 and the same processes are repeated. When the process 

21 reaches the last node, the program steps to S207 where the node 

22 having the smallest reference number (the node nearest to the 

23 self vehicle) and the node having the largest reference number 

24 (the node furthermost from the self vehicle) , are searched 

25 respectively and leaves the routine after denoting them as a start 

26 point N s and an end point N e , respectively. 

27 After this program is carried out on the side wall 

28 groups on the left side, it is carried out on the side wall groups 
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1 on the right srcle . In the example shown in Fig. 14, the wall surface 

2 from the 9th node to the 26th node is detected on the right side 

3 of the self vehicle and the 9th node is denoted as the start point 

4 N s and the 2 6th node is denoted as the end point N e . These nodes 

5 are used for later processes as effective nodes. 

6 Thus processed position of the wall surface is further 

7 corrected by a program shown in Fig. 7 and Fig. 8 using new data 

8 obtained from programs shown in Fig. 3 through Fig. 5. 

9 The programs shown in Fig. 7 and Fig. 8 is a program 

10 for correcting the wall surface. At S301, it is investigated 

11 whether or not the start point N a of the effective nodes is larger 

12 than the first node Ni of the wall surface model. When N 8 - Ni, 
O 13 the wall surface has been already detected up to the first node 
IU 14 Ni, the program skips to S306. When N a > Ni, the program goes to 

CO 15 S302 where the previous node N 8 -i (i-1, 2, 3 etc . ) is es tabl ished . 

m 

ry 16 Then, at S303 the wall surface pattern is searched and at S304 

Ur. 17 the X- coordinate of the wall surface is determined based on the 

ry 18 result of searching. 

19 Next, the program goes from S304 to S305 where it is 

20 investigated whether or not the process has reached the first 

21 node. If not yet reached the first node Ni, the steps S302 to S304 

22 are repeated to continue the searching of the wall surface 

23 position up to the node Ni. When the processes up to the first 

24 node Ni are finished, the program goes to S306 where it is checked 

25 whether or not the end point N e of the effective nodes is smaller 

26 than the last node N ae of the wall surface model (for example, 

27 node N 4i in case of the wall surface model constituted of 41 nodes) . 

28 As a result of this, when N e - N ae , that is, the wall 
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1 surface has bUh already detected up to the last node, the program 

2 skips from S306 to S311. When N e < N ae , the program goes from S306 

3 to S307 where the node N a+ i after the end point N e is successively 

4 established and further at S308 the pattern matching of the wall 

5 surface is performed. According to the result of the pattern 

6 matching, at S309 the X-coordinate of the wall surface is 

7 determined and then at S310 it is checked whether or not the process 

8 has reached the last node N Be . The matching of the wall surface 

9 position is continued until the last node N se and when the processes 

10 up to the last N ae is finished, the program goes to S311. 

11 These processes of establishing the nodes, the matching 

12 of the wall surface pattern and the determination of the X- 

13 coordinate at the steps S302 to S304 and the steps S307 to S309, 

14 are the same as the processes at the steps S203, 204 and S205 

15 in the aforementioned program of the wall surface detecting 
e 16 process . 

; 17 The processes after S311 are for correcting the 

1H 18 position (X-coordinate) of respective nodes from the first node 

5 19 Ni to the last node N 8e . First, at S311 the data of the first node 

H 8 20 Ni is set and the program goes to S312. The processes from S312 

21 to S321 are repeatedly carried out by successively setting the 

22 data of the next node. 

23 At S312, the wall surface at the node Ni is searched 

24 and at S313 it is checked whether or not the wall surface is 

25 detected by the pattern matching. If it is judged that the wall 

26 surface is detected, the program goes from S313 to S314 where 

27 it is investigated whether or not the difference between the 

28 position X pw of the wall surface and the position X n i of the node 
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1 is within a prescribed amount, for example, ± 1 meter. If the 

2 difference is within the value, the program goes to S315 where 

3 the node is moved to the position of the wall surface (X ni <- x P „) 

4 and if the difference is out of the value, the program goes to 

5 S316 where the node is moved toward the wall surface by a specified 

6 amount, for example, ± 0.3 meters (X ni <- x ni ± 0.3 meters). 

7 On the other hand, when the wall surface is not detected 

8 by the pattern matching, the program diverges from S313 to S317 

9 where the number C 0 of the data X P i of the solid objects located 

10 on the left side of the node X ni (X ni < X pi )and the number d of 

11 the data X pi of the solid objects located on the right side of 

12 the node X„i (X„i > X p i) , are counted respectively. Then, at S318 
O 13 the node is moved towards the side having more data of solid objects 
fy 14 by a specified amount, for example 0.8 meters (X n i <— X„i±0 . 8 
Oj 15 meters) . 

fU 16 Thus, in case where there is no wall surface pattern 

!^ 17 detected near the node, the position of the node can be neared 

pj 18 in the direction where the wall surface likely exists even when 

S 19 the node is largely apart from the wall surface. 

20 When the position of the node is moved through either 

21 step of S315, S316 and S318, the program goes to S319 where a 

22 position X c of a mid-point of a straight line connecting a node 

23 Ni +1 (adjacent node on far side) and a node Ni-i (adjacent node on 

24 near side) is obtained and then goes to S320 where, as shown in 

25 Fig. 18, the node Ni is moved toward the mid-point X c just like 

26 in a manner that the mid-point X c attracts the node Ni by a spring 

27 force. The amount of the movement should be retained as much as 

28 1/2 to 1/5 of the length between the node Ni and the, mid -point 
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1 x c . . 

2 That is, the wall surface detected by the pattern 

3 matching generally contains irregularity due to the effect of 

4 the dispersion of data but a real guardrail or real side wall, 

5 in most case, is smoothly curved along a road. The process at 

6 S320 is a means for obtaining a smooth curve by applying a spring 

7 operation as described above. As a conventional method of 

8 smoothing the configuration composed of nodes, a least square 

9 method is well known, however the aforegoing method using a 

10 spring operation is more advantageous in calculation speeds than 

11 such a conventional method. 

12 After that, the program goes to S321 where it is 
Q 13 investigated whether or not the process has reached the last node 
fy 14 N se and if it has not reached the last node point, the program 
m 15 returns to S321 after setting the data of the next node at S322 
fy 16 and the same processes are repeated. When the process at the last 

s 

m 17 node N se is finished, the program goes from S321 to S323 where 

nj 18 it: is checked whether or not the amount of the movement comes 

gl9 within a threshold value (for example, ±0.1 meters) for all of 

2 0 the points. 

21 Further, if a node exceeding the threshold value is 

22 found, the program goes back to S311 and thereafter the correcting 

23 processes are repeated for all nodes from the first to the last. 

24 When the amount of the movement for all node points comes within 

25 the threshold value, the start point N fl and the end point N e within 

26 the detecting range of the nodes are obtained and the program 

27 leaves the routine. Through thus constituted program, erroneously 

28 detected data and others are corrected during repeated processing 
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1 and as a result the configuration of the wall similar to the actual 

2 wall surface is obtained. After the program is finished to be 

3 carried out with respect to the side wall groups on the left side, 

4 the program is carried out with respect to those on the right 

5 side. 

6 Fig. 19 shows a result of the detection of the wall 

7 surface located from near to far along a curved road on the basis 

8 of the original image shown in Fig . 9 and the corrected wall surface 

9 model expressed on a X-Z plane is shown in Fig. 20. Comparing 

10 this wall surface model with the one detected in the form of 

11 straight lines as shown in Fig. 13, it is understood that the 

12 wall surface can be recognized up to far, 

13 This wall surface model can be applied not Only to a 

14 guardrail but also to a line of trees, walls of houses, an array 

15 of parked cars and other solid objects lined along a road and 

16 these objects can be detected as a series of wall surfaces. 

17 Therefore, it is possible to recognize the configuration of a 

18 road even in case of a road whose lane markers can not be recognized, 

19 for example a snow- covered road. 

20 While the presently preferred embodiment of the present 

21 invention has been shown and described, it is to be understood 

22 that this disclosure is for the purpose of illustration and that 

23 various changes and modifications may be made without departing 

24 from the scope of the invention as set forth in the appended claim. 
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